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In order to evaluate the validity of outcomes 
assessment at two-year colleges, the American College Testing (aCT) 
Program developed "Project Cooperation." Institutions participating 
m the project administered tests and surveys to measure changes in 
students' cognitive abilities over time and record student feedback, 
and reported the data along with curricula and student transcript 
information to ACT. In addition, ACT developed an achievement index 
and a planning/test content form to help institutions develop the 
best research design and match course objectives with test 
objectives. To measure cognitive outcomes, 78 colleges applied the 
Collegiate Assessment of Academic Proficiency (CAAP) test to freshmen 
in 1989 and 1990 and again to the same students in 1992. To gather 
student feedback, 72 participating institutions administered the 
College Outcomes Survey (COS) in spring 1992. The COS includes 
sections on student background, college outcomes or goals, student 
evaluation of the importance of each outcome/goal, student progress 
in all areas, and student satisfaction. While many institutions 
failed to achieve the minimum number of 100 matched records of 
students* cognitive outcomes and transcripts, results of the 
assessment of student feedback included the following: (1) acquiring 
knowledge and skills in an area of specialization ranked as the 
highest goal and as the area of most progress; (2) students responded 
positively towards colleges' general education programs; and (3) the 
areas of highest satisfaction were class size and response to older 
aad nontradit ional students. Extensive appendixes provide the CAAP 
planning form, principles for use of the CAAP in outcomes assessment, 
the COS, and the full Project Cooperation Community College Composite 
of the College Outcome Survey Report. (MAB) 



o 

CO 

Q\ Student Outcomes Assessment: Are. We as Good as We Think? 

CO 



David A. Lutz , Ph.D 
Director of College Level Assessment & Survey Services 
American College Testing 
Iowa City, Iowa 



(J 

0 

o 

CO 

cr 

ERiC 



PERV.iSS ON TO =^EfJ»CDuCE: -HiS 
MATEfsiAL HAS BEEN '}>P^\^tZ 

D. A. Lutz 



December 18, 1992 



iNPORMATlON v" t "f^ R . E PttC > 



U.S. DEPAin-MENT OF EtXJCATION 
0«»ce of Educational Res««rch «nd Impfovement 

EDUCATtONAL RESOURCES INFORMATION 
CENTER (ERiCi 

C This <Jocumt.it has been reproduced as 
received from the oerjon c organizatior^ 
originating it 

Minor changes have been made to impfove 
reproduction quality 



Pomts oi view or Opinions stated m m»s oocu' 
ment do not necessarily represeitt o'ficiai 
OERl position Of policy 



STUDENT OUTCOMES ASSESSMENT— ARE WE AS GOOD AS WE THINK? 



David A. Lutz, Ph.D. 
Director of College Level Assessment & Survey Services 
American College Testing 



At first blush we want to answer "Yes," basing our response on our best intuitive feelings 
and our beliefs that at our institution we are good. However, this answer by itself is not 
considered adequate. Empirical evidence as derived from valid research is required if our 
answer is to be accepted by external bodies (e.g., the federal government, state 
goveniment, accrediting agencies) demanding accountability and the various publics we 
serve. In addition, the issue of institutional effectiveness has been and continues to be an 
important one from the perspective of the American Association of Community Colleges. 
Further, if an institution really wants to do the best job possible, it must document student 
achievement and be prepared to make changes in programs where required as a result of 
good research data. The important question is, how can we validly demonstrate our 
effectiveness? 

This question was a major stimulus for a national research project — Project 
Cooperation — that began in 1988 as a result of an initiative from the American Association 
of Community Colleges. Subsequently, two major councils of AACC, the National 
Council for Instructional Administrators and the National Council for Student 
Development, and American College Testing joined efforts to cooperatively develop 
assessment models, both cognitive and affective, that can be emulated by other community 
colleges. To carry out this research, 12 institutions/systems were initially involved. These 
are: Midlands Technical College, Orangeburg-Calhoun Teclinical College, Technical 
College of the I.owcountry, Mass Bay Community College, Howard Community College, 
Macomb Community College, the St. Louis Community College System, the Metropolitan 
Community College System, Scottsdale Community College (Maricopa System), Riverside 
Community College, and Chemeketa Community College. 

Cognitive Perspective 

Several models were proposed, but most institutions agreed to pursue change-related 
cognitive research. This research is longitudinal, that is, incom.ing students are assessed to 
establish a baseline and these same students then are assessed upon exit near the end o*' the 
sophomore level to show change. This type of research is the same as the I-E-0 Model 
presented by Dr. Alexander Astin in his book. Assessment for Excellence. Entry level 
testing was carried out in 1989 and 1990 using ASSET (a major placement test for 
community colleges) and/or the Collegiate Assessment of Academic Proficiency or CAAP 
(an outcomes instrument designed to measure foundational skills achievement at the end of 
the sophomore level). It was anticipated that many of these students would have 
completed at least 45 semester credits by the spring of 1993 when CAAP would be 
administered to these same students. This design would allow for a pure longitudinal 
study cohort using CAAP as the pretest and CAAP as ^\c posttest. It would also provide 
data for change study using ASSET as the pretest and tlie similar hut more advanced test, 
CAAP, as the posttest. The purpose of this second design is to establish a reliable mclhod 
of assessment that reduces the need for excessive testing and its costs. This approach 
might positively affect student motivation as well. 



In 1991 ACT staff developed an achievement index methodology that would enable 
institutions to infer academic change over time, based on students' perfonnance on two 
sequentially or construct-related tests. To validate tJiis methodology, a minimum of 5000 
matched test results (ASSET and GAAP) would be needed for each subject: writing 
skills, reading, and mathematics. Additional institutional participation wa.s necessary if we 
were to obtain the required numbers of matched test results. 

To address other concerns related to outcomes assessment success, ACT staff designed a 
GAAP Planning Fomi and Test/Content Form (see Appendix A) to help institutions 
develop the best research design and to help tliem match course objectives witli test 
objectives. This research will not produce the desired data unless students tested with 
GAAP take the related courses. Faculties at each instituMon were to analyze their curricula 
to detcnninc tiie match between test content and courses. In addition, we required selected 
transcript information to document the courses taken by the GAAP-tested students. See 
Appendix B for a detailed expkuiation of the overall research design. 

Invitations were then sent in the fall of 1991 to several hundred community colleges across 
the country. Institutions invited to participate in the expanded research were on record as 
purchasing substantial numbers of ASSET (Forms B/G) in 1989 and 1990, assuming that 
many of their students would have met the minimum requirements for the design for 
sophomore level testing by spring of 1992. The requirements were that students took 
either Form B or G of ASSET at least one year earlier, that they had completed between 
45 and 75 semester credits (although this number was later changed slightly), and that 
most of their credits had been earned at the institution agreeing to participate in tliis 
research. These institutions were ta administer at least two GAAP modules to a minimum 
of 100 students each in the spring of 1992. Each institution could select the GAAP tests 
appropriate to its needs and could administer the tests in a two-week block of time of iis 
own choosing. Each institution was also to provide the relevajit student transcript data to 
ACT before the end of the summer. 

Seventy-eight institutions agreed to participate as Project Cooperation "Affiliates" and 
submiued their research plans to ACT for review. Most institutions chose to administer 
GAAP in classrooms rather than testing a random sample of students at a central testing 
site. This metJiod was preferred by many institutions because it wds viewed as the 
simplest way to address the motivational problem of getting students to test. On the other 
hand, it created problems because students selected in this manner may not be 
representative of all sophomores, particularly if a large portion of the students in a class do 
not satisfy the research criteria. This method also did not solve the problem of students 
doing their best on the test(s). 

GAAP testing commenced in late February of 1992 and ended in June. Several institutions 
were successful in testing a substantial number of students. Many, however, fell 
significantly short of the required numbers with overall attrition rales (differences between 
the number of modules ordered and the number administered) ranging from 11% to l()()7r. 
Student transcript infonnation was then prepared by most of tlie institutions over several 
months, but some infonnation was not sent to ACT until mid-October. Throughout the 
fall, ACT staff did everything possible to get the best match of test scores and student 
records. Unfortunately, the end result was that few institutions had attained the minimum 
of 100 matched records per subject, (a minimum of 100 students is needed to establish an 



achicvemcnl index for individual institulions to obtain meaningful information.) Thus the 
goal of developing an achievement index was not attained. 

Matching student records for this complex research presents significant challenges, 
particularly with respect to collecting die data. Let me illustrate how difficult this process 
is. For this project, orders were placed for approximately 15,000 Writing Skills modules. 
Of these, 7,028 were actually administered witli 1902 having preliminary matche,^ Final 
matching efforts (ASSET Forms B/C, GAAP lest results, student transcript information, 
plus the other criteria referenced above) produced 785 matched student records. This 
means that only 11% of the original number tested produced complete matches for the 
research pertaining to documenting writing skills change. If one considers tlie numbers 
originally planned for, this number is reduced to 5%. To successfully compicte change 
research using multiple data sources, an institution should assess all students completing a 
program of study. This is bhc only way to obtain sufficient infoniiation for decision- 
making. Obviously the numbers needed to answer our question are currenUy too small to 
let us know if we are as good as we think we are from the cognitive perspective. 

Because the achievement index system has already been proven for use witli otlier ACT 
programs, it is conceivable that this method would work with GAAP ajid ASSET. 
However, colleges must collect more data until the required numbers are met to establish a 
national reference group for achievement index reporting. ACT will continue to work with 
institutions until this goal has been met and then will offer this service to those willing to 
provide llie required data. Further, ACT has developed an alternative reporting method 
that is less complicated and allows an institution to document student achievement widi a 
minimum of 100 students per subject area. (A minimum overall total of 5,000 students 
per subject area is not required to establish a proper reference group using this alternative 
approach.) This type of reporting will be av:iildble later this year when individual 
institutions have tlic data. 

Before turning to the affective perspective on this question, some observations should be 
made that may be helpful to institutions researching academic chani^e. These observations 
are as follows. 

1. Strong public commitment lo improve the institution is required by the president 
and the chief administrative officers. 

2. Extensive faculty support is absolutely neo ssar>' if the research is to be successful 
and the results used to improve programs. * 

3. There must be an institution-wide public commitment to outcomes assessment. 

4. Adequate lead time is critical to get the model in place and implement it 
successfully. 

5. Detailed written plans should be prepared ajid agreed to before proceeding. 

6. Students should be required to participate in outcomes assessment as a routine part 
of the collegiate experience mid Uiis requirement must be published in the college 
catalog and other documents. 

7. Students should be fully apprised of the rationale for the assessment effort and 
shown that their participation and data obtained will be used by the institution for 
improvement. 

8. The person(s) assigned the responsibility for the assessment effort must have the 
authority to do the job. 

9. Multiple relevant data sources must be used to miswer each outcome question. 
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10. Baseline data must be obtained for all incoming students. 

1 1. Student sampling methods must be large enough to produce the desired number of 
matched records. 

12. The results must be used to improve programs where warranted. 

If community colleges utilize these suggestions for change research, they will have a 
greater likelihood of successfully documenting student academic achievement and will 
show their constituents and external boards and agencies just how good they rejOly are. 

Affective Perspective 

The answer to the question, "Arc we as good as we tliink?" can be more readily obtained 
from student opinion feedback. We were able to obtain good data from the second major 
component of Project Cooperation research through the College Outcomes Survey. This 
instrument was developed by ACT survey experts witli extensive input from tiie National 
Council for Student Development and Project Cooperation institutions. This optically 
scanned instiument is four pages in length and takes about 30 minutes to complete. See 
Appendix C. It consists of several parts: background information, college outcomes or 
goals with student evaluation of the importance of each and the progress made in attaining 
it, feelings about general education at the college, general evaluation of the college, 
personal growth in a number of non-academic areas with an evaluation of tlie college's 
contribution to that growth, student satisfaction with given aspects of the college, a section 
for the college to ask up to 30 locally-developed questions, and space for comments and 
suggestions. 

The College Outcomes Survey was administered by 72 Project Cooperation community 
colleges in the spring of 1992. A report, based on 9,557 student surveys from these 
institutions, was prepared and published. A free copy of tliis report may be ordered 
through the author at ACT. This report provides aaswers to our basic question. The 
colleges represented in the report include many different sized institutions: \29( from 
colleges with enrollments of over 10,000; 179^ from colleges with enrollments of 5,{X)() to 
1(),(XX); 509^ from colleges with enrollments of 2,000 to 5,000; aiid 2\% from colleges 
with enrollments under 2,000. All are public community colleges located in 24 states. 

Before summarizing the results, several qualifications are needed. The data are not based 
on a random sample of students or colleges. The instruments were administered in 
different ways to different groups of students (thus the response rates varied widely). The 
number of cases and institutions in this report is limited. Thus comparisons based on the 
data must be interpreted with caution. 

Background summary: Females represent 63% of the respondents. The average age is 
26.43, with about 46% being 21 or below. Approximately 87% arc Caucasian, with 
African- American being the largest single minomy group (6%). Seventy-nine percent have 
overall college grade averages of B- or above. The cunrent course load of these students 
is: 219( are taking 9 or less credits and 65% are taking 10-18 credits. The total number 
of credits that will he earned hy the end of the tenn are as follows: 20% have under 36 
credits, 21% have between 36 and 60 credits; and 32%> have between 60 and 83 credits. 
Almost three fourtlis of the students have earned all their credits at the college they are 
cunrently attending. Another 14%. have transfenred from 1-15 credits. The major areas of 
sludy represented by the survey-takers in the order of most students to least students arc: 



health science - 26%. business and management - 179f , education - 6%, social science - 
5%. undecided - 59r, pre-engineering - 5%, community services - 5%. teacher education - 
47r, computer science - 4%. business and office - 4%, engineering - 4%, trade and 
industrial - 4%, etc. SHghtly more than a third plan to transfer to a four year college the 
next academic year, 37% plan to re-enroll in the same college, and 13% do not plan to 
attend college. 

Several responses pertain to educational achievement and goals. About half have earned 
no degree or certificate since high school and 28% have earned an associate degree. Wlien 
these students enrolled tlic lifetime goal was an associate degree (for 27%), a baccalaureate 
degree (32%), master's degree (18%), and doctorate/professional degree (9%). These 
percentages have ciianged since enrollment in the college. They now are: associate degree 
(14%), baccalaureate degree (31%), master's degree (30%), and doctorate/professional 
degree (12%). What is significant here is that the community college experience has- 
caused many students to raise their educational goals to a higher level. This is one of 
many positive outcomes reflected by the results of this survey. 

The last background question pertains to students' responsibilities and time allocations. 
Let me highlight a few areas. Twenty-nine percent spent 21 or more hours per week in 
course-related activities. Less than a quarter (24%) are not working for pay; 27% arc 
working over 30 hours; 16% arc working 21-30 hours; and 13% arc working 16-20 hours 
per week. Only 29% are not involved in care of family. About 27% are spending 30 or 
more hours caring for family; 12% spend 16-30 hours; 18% spend only 1-5 hours. 

Importance of and progress toward attaining outcomes at this college: Average 
ratings of each outcome statement (i.e., for each of 26 items) were ranked first in tcmis of 
the level of importiuicc to the student of attaining the outcome, and then in tcnns of the 
amount of progress attained on each. For the importance ratings, a 3-point scale ranged 
*>cmi "Of great importance" (3.{X)) to "Of little or no importance to me" (1.00). For the 
progress ratings, a 3-point scale ranged from "A lot of progress" (3.00) to "Little or no 
progress" (1.00). 

On tlie importance scale, the following ten outcome statements received the highest 
average ratings: acquiring knowledge and skills in my area of specialization (2.94), 
acquiring knowledge and skills needed for a career (2.87). improving my ability to make 
better decisions (2.77), leaming to set goals and follow through to completion (2.77), 
learning to think and reason (2.72), understanding my strengths and weaknesses (2.71), 
improving my ability to apply new information (2.66), improving my study skills (2.64), 
listening to and understanding what others say (2.63), and developing problem solving 
skills (2.63). On the progress scale, the ten highest average ratings were as follows: 
acquiring knowledge and skills in my area of specialization (2.44), acquiring knowledge 
and skills needed for a career (2.41), learning to think arid reason (2.38), leaming to .set 
goals and follow tlirough to completion (2.37), understanding my strengths and weaknesses 
(2.34), improving my ability to make better decisions (2.33), listening to and 
understanding what others say (2.32), improving my ability to apply new infonnation 
(2.31), thinking objectively about beliefs, attitudes, and values (2.29), and developing 
problem solving skills (2.23). 

Although the average rating for a given item on the progress scale generally fell about .5 
to ,4 points below the average rating for that siunc item on the importance scale, it is 
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important to remember that the two scales are different and should be interpreted in tenns 
of the meanings associated with points on each. Nevertheless, the order of items remained 
relatively close to tlial found on the importance rankings. Bear in mind that the mid-point 
(2.00) on each 3-point scale represents "average or moderate" on each of the four rating 
scales of importance and progress here and growth and college contribution referenced in 
the persona] growth section below. When a rating moves above the mid-point on the 
scale, i.e., above a "moderate or average" rating, we can assume the colleges v*re perceived 
by respondents as doing an above average job. By the same token, when the average 
rating falls below the mid-point we want to take notice. A few ratings did fall slightly 
below the mid-point on the rating scale. For example, three areas in which students 
reported the least progress on average were in increasing appreciation of art, music, 
literature and humanities (1.87), improving physical coordination, dexterity, and muscular 
or motor skills (1.87), and developing original ideas or products (1.92). The first two of 
these were also rated lowest in importance, but the third, developing original idea.s or 
products, was rated somewhat above the mid-point (2.35) on the importance scale. 
Perhaps community colleges should take a closer look at how they arc serving students 
who place importance on developing original ideas or products. 

Feelings about this college's general education: The students were quite positive in their 
feelings about the college's general education program, holding that it will benefit them in 
their personal/professional life and that it helped them develop skills in English, math, 
social sciences, natural sciences, and the humanities. They do not feel tliat general 
education requirements are a waste of their time. 

(ileneral agreement with statements about the college: Overall, students arc satisfied 
with the college as a whole. They arc proud of their accomplishments at the college 
(agreement average of 4.25 on a 5 point scale, where 5 = strongly agree and I = strongly 
disagree) and would recommend the college to others (4.13). They feel that the college has 
helped them meet their goals (4.09). The lowest level of agreement was the item, the 
college welcomes and uses feedback from students to improve the college (3.62). This 
rating shows that the students feel this issue falls between agree and neutral, neither agree 
or disagree. It should be noted that if students perceive the college docs not welcome tlicir 
feedback, the college will have a very difficult time getting tlicm motivated to participate 
in assessment activities because they believe the college won't use the data to improve 
programs and services. Institutions must show students that tliey will use student feedback 
for improvement if they arc to get students to participate and do their best. 

Personal growth since entering college: Students were asked to relate 32 outcomes 
statements using two separate 3-point scales, the first referring to the extent of personal 
growth they had made since entering this college, and tlic second referring to the extent of 
the college's contribution, both in and out of class, to personal growth. On the first of 
these two scales, the points were defined as follows: "A lot of growth" = 3; "Moderate 
(average) growtlV = 2; and "Little or no growUi" = 1. "Not a goal of mine" was a fourth 
option, but this response was not included in the calculation of personal growth averages. 
On the report prepared for colleges, tlic average growth ratings for each item were ranked 
from highest lo lowest, with the 10 highest average ratings being as follows: setting a 
direction for my life (2.52), increasing my intellectual curiosity (2.44), becoming 
academically competent (2.42), implementing long-tenn or life goals (2.41). taking 
responsibility for my own behavior (2.41), developing self-confidence (2.41), improving 
my ability to relate to others (2.36), making a life-long commitment to learning (2.35), 
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increasing self-understanding (2.35), aiid becoming more willing to consider opposing 
points of view (2.28). 

The six items witli the lowest average ratings were the following: becoming more aware 
of local, regional, and international issues and events (2.06), maintaining my physical 
fitness (2.02), understanding religious values differing from mine (2.01), learning tiie role 
of volunteering to support worthwhile causes (1.97), and preparing myself to participate 
effectively in the electoral process (1.91). It is interesting to note that students in 
community colleges gave only "moderate" ratings to items such as "dev;lop good physical 
fitness", "understanding religious values differing from my own," and "preparing myself to 
participate effectively in the electoral process." 

On tlie other hand, students report above average growth in a number of areas in which 
one would hope for such growtii. The question here is, how much has the college 
contributed to this growth botli in and out of class? On the 3-point college contribution 
scale, the top ten outcomes were as follows: becoming academically competent (2.31), 
increasing my intellectual curiosity (2.30), making a lifelong commitment to learning 
(2.18), setting a direction for my life (2.17), implementing long-term or life goals (2.16), 
developing self-confidence (2.14), learning to critique and judge infonnation (2.13), 
improving my ability to relate to others (2.11), become more willing to consider opposing 
points of view (2.05), and becoming an effective team or group member (2.05). All in all, 
community colleges are doing a good job in many areas where they should be expected to 
contribute to student growth — both academically and in the affective domain. 

Satisfaction with given aspects of this college: Finally students were asked to evaluate 
how satisfied they are with a variety of aspects of tiie college. On a 5-point scale where 
5=very satisfied and l=very dissatisfied, greatest satisfaction is found in the following: 
class size (4.1 1), college response to older/nontraditional students (4.08), quality of my 
program of study (4.05), quality of instruction (4.05), freedom from harassment on campus 
(4.03), college response to students with special needs (3.94), library/learning resources 
center services (3.93), faculty respect for students (3.92), student access to computer 
services and facilities (3.90), and availability of faculty for office appointments (3.86). 
Most of tiiese outcomes are related to tiie quality of the academic programs and the 
commitment of tlie faculty to teaching. 

Even in the areas of least satisfaction, students tended to he satisfied (i.e., their average 
ratings were well above the neutral rating of 3.00). For example, the aspects with (he 
lowest satisfaction ratings were the following: student health/wellness services (3.50), 
personal counseling services (3.50), language development services lor students whose first 
language is not English (3.49), recreational and intramural programs (3.47), job placement 
services (3.47), veterans services (3.39), mental health services (3.31), residence hall 
services and programs (3.30), support services for victims of crime and harassment (3.26). 
These are areas that community colleges might investigate to detemiine how students' 
needs can be served better. It is possible that many of these services have been affected in 
recent years because of shrinking budgets. It is also possible that these are areas that have 
traditionally been of lower priority at community colleges. Generally spcaknig, 
community colleges are doing a good job overall in meeting students' needs from the 
perspecUve of tlieir students. 
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It is apparent from the above summary, based on tlie data obtained from students 
completing the College Outcomes Survey, that community colleges overall are doing a 
good job. They are positively affecting student outcomes that should be expected from tlie 
mission and objectives of community colleges. NeverUieless, there are many areas that are 
not as good as they should or could be. These areas are readily apparent when one looks 
at the data from individual colleges. The College Outcomes Survey is sensitive to these 
shortcomings and readily shows where problems may exit. We need to investigate these 
further, considering data from several sources, and Uien reach conclusions regarding what 
must be changed. 

Conclusion 

Student outcomes assessment is a valuable method to determine how effective community 
colleges really are. To benefit from this approach a strong institution-wide commitment is 
required. Once tliis commiunent is evident, students will be motivated and will give us 
the data we need. We then can document how much we really help our students achieve 
their academic and other goals. We can show that we really are as good as we think. 



December 18, 1992 
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GAAP Outcomes Assessment Planning Fomi 

Thorough and precise planning is essential for effective outcomes assessmer*. By considering carefully 
issues such as the correspondence between course content and test content, the selection of students for 
testing, student motivation, and tost admirtstratlon, an institution can be more confident of obtaining 
meaningful assessment results, and of optimally utilzing staff time and financial resources. 

This fonn is designed to assist you in planning your outcomes assessment Although elaborate or detailed 
responses are not necessary, we encourage you to respond thoughtfully. We also encourage you to read 
the document entitled "Prindpies for Effective Use of GAAP in Outcomes AssessmenT before comqietinq 
thefomt ^ 

Instructions: 



1. Pfease complete Parts I and II. 



Send your completed fom to ACT as far in advance of testing as possible so that ACT staff can 
review it and malce suggestions if necessary. The form should be sent to ACT GAAP 
Operations, P. 0. Box 168, Iowa Ctty, lA 52243. 

If you have questions or need assistance, please contact either of the following individuals: 

David A. Lutz, Ph.D. jeff Schiel, Ph.D. 

Director Research Associate 

College Level Assessment Services Research and Statistical Services 

31 9/337-1 051 31 9/337.1 076 



Part {. 



Please fill in the information below. 



Your name 



Trtle 



Institution 



Address 



Telephone f ) Extension 

Best tirnr^ to 

be contacted . 




ERLC 



1: 



PartiL 



1. Please list the assessment questions that are of interest to your institution, and the data sources 
that you believe can help provide answers to these questions. An example of an assessment 
question might be: "Does our general education core curriculum contribute to an increase in the 
skills and knowledge that we want our students to acquire?" Examples of relevant data sources 
could include courses taken, course grades and GPAs, and GAAP test scores* 

AMMsmMt quMtksns Data aourew 



1 ^ 



2. Please describe how you plan to use the results of your outcomes assessment study. What 
actions will you take, depending on the results? 
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Pfeasa describe the prindpai foundational skills your institution seeks to develop in its general 
education core program. Thon. next to each skill, list the prindpal core course or courses in which 
these skills are developed. Rnaliy, indicate the GAAP test(s) you intend to use to assess students' 
achievement in these skills. 



GAAP may not be appropriate for measuring ail of your institution's foundational' skills. If it is not 
appropriate, please indicate this as well. 



Here are some examples of foundational skills, courses, and GAAP tests: 




Ypu 1^ thte •piwj.ch or th* T««tCoirtint 


Exempiea 




SkJJ! 


Courae(e^ 


CAAP temtfs) 


D0wongtrai9 Gonv&ntionaJ sppJication of punctusnion, 
grmm^r^ and sanmncm struetum. Drnnonsmw tha 
9biiitf to OTQunizm id9Mz. 


English Composition (ENG 100) 


Writing Skills test 


Dmnonsomm aa understanding of ihm scmntr^c method 
end tJi9 cafmdty to evaJuete compming hypotJietms, 


Scientific Inquiry (SC 120) 


Science Reasoning 
test 


Dmnonetratm thm abiSty to praaant a waU^miizad 
yerbai report andfor spaacJu 


Public Speaidng (PS 100) 


(not aopropnate) 



The contents of the CAAP tests are described briefly in an attachment to this document. Please 
read the attachment before completing this question. If you would like more detailed information 
on the test contents, please see the CAAP Technical Manuoi. 



Skill 



Course(s) 



CAAP test(s) 



Please describe the students who will be the focus of this study (the "reference group"), and how 
they relate to the general education core program. (Many institutions define the reference group 
as students wha have earned between 4S and 60 credits at the beginning of the semester in which 
they are tested.) 



Please describe any sul^groups of the mair reference group that you warrt to study separately. 
(Exampies: Engineering major? Hispanic students, non-traditionaJ age students.) 



3. (continued) 



Skill Course(s) GAAP test(s) 



(Please attach additional pages if needed.) 



Please state below whether you plan to test the entire reference group, or whether you intend to 
test a represent^itive sample from it. If you plan to test a representative sample, please describe 
how you will select the sample. (Examples of acceptable methods of sampling are provided in the 
"Principles" document. ACT recommends a minimum sample size of 100 students per reference 
group.) 



If you plan to study a spedaJ subgroup, state whether you plan to test all students in it, or whether 
you intend to test a representative sample. If you plan to test a representative sample, please 
describe how you will select the sample. Note: Selecting members of a special subgroup from 
a representative sample of the entire reference group may not result in sufficient data for analysis. 



Please describe your plans to inform your tacufty and to enlist their support for the study. 



Please describe how you plan to motivate students to partidpate in the study and to do their best 
when taking the GAAP. (Suggestions for motivating students are pr^,Yided in the "Prindples** 
document) 



10. Briefly describe your plans for adminis^9ring the GAAP, including test dates. 
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A Brief Sumsnaiy of the 
Contents of the GAAP Testa 



Wrttlng Skills Test— Measures students' comprehension of punctuation, grammar, sentence 
stnjcture, and other elements necessaiy for effective written Engfish. 

Reading Test^-Measures reading comprehension by requiring exan^nees to refer to explicit 
statements and then reason, draw conclusions, and generalize beyond the written material. 

Mathematics Test—Measures mathematicai si<ills in content areas ranging from pre-aigebra to 
introductory calculus. 

Critical Thinictng Test---Measure3 students' sldlis in classifying and analyzing the elements of an 
argument, and in evaluating and extending an argument 

Science Reasoning Test— Measures scientific reasoning sidlts, but does not emphasize factual 
knowledge. Students are required to interpret graphs, tables, and scatter plots, analyze 
experimental results, and compare alternative hypotheses or viewpoints* 

Writing (Essay) Test— Measures students' skills in formulating and supporting assertions ;ut 
a given issue, and in organiang and connecting major ideas. 



For more detailed Infomiatton regarding the 
contents of the CAAP tests, please see the 
CAAP Technical Manual. 



GAAP Test Content Form 



An important step in ensuring that assessment results will be meaningful is for an institution to identify 
logical relationships among its goals for developing students' fundamental skills and knowledge, the 
contents of the courses in its general education core cumculum, and the contents of the GAAP tests. 

For example, if the institution wishes to ensure that all of its graduates are proficient in writing 
grammatically con-ect and logtoally coherent essays, then the requisite skills and knowledge must be taught 
through specific courses in the core cumculum (e.g., freshman English composition). The institution could 
then consider measuring these skills and knowledge with relevant GAAP tests; in this instance, the Writing 
Skills and Writing Essay tests would be appropriate. 

This fomi is designed to help institutions identify relationships among general education goals, core 
courses, and the contents of the GAAP tests. Because a single individual is unlikely to be familiar with his 
or her institution's entire general education cumculum, ACT recommends tnat several faculty members, or 
pertiaps a general education committee, collaborate when completing this form. Mathematics faculty, for 
example, are familiar with the contents of core courses in mathematics, and could therefore determine 
whether the GAAP Mathematics test will measure the skills and knowledge taught in these courses. They 
may not, however, be familiar with the contents of core courses in English. 

The principal use of this form, for most institutions, will likely be to identify GAAP tests that are appropriate 
for measuring students' fundamental academic skills and knowledge. It is possible, however, for the form 
to sen/e additional purposes. For example, the form may initiate discussions of general education goals 
In relation to the core courses currently being taught. Gollaborating faculty then could determine that a 
general education goal might be achieved more readily by consolidating certain courses or by adding other 
courses to the core cumculum. 

Instructions: 1. Please fill in Sections A, B, and (if applicable) Section C. 

2. Mail the completed form to: Gollege Level Assessment Sen/ices 

American Gollege Testing 
P. O. Box 1 68 
Iowa Gity, lA 52243 
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7/12/91 

Principles for Effocttvo Us« of GAAP In Outcomos Assessment 



Postsecondary institutions are being asked, with increasing frequency, to provide evidence of the 
effectivensss of their oducatbnal programs to legislatures, state boards, and other goveming and funding 
bodies. There is a demand for institutions to demonstrate that students gain skills and knowledge by 
participating in postsecondary educatton. "the results of such outcomes assessment efforts often receive 
considerable attention, and it is not uncommon for the results to be tied to program development and even 
to funding decisions. 

Institutions themselves often conduct self-studies, irrespective of external accountability requirements. The 
results of such studies can be important in making decistons about program devekjpment, continuation, or 
enlargement, in detecting and ameliorating problems, and in how the institution perceives itself. 

To initiate assessment studies, institutions must address several issues. They will need, for example, to 
devetop formal plans, identify partfcipants among administrators, faculty, and students, and select 
appropriate research designs. Outcomes assessment can be a large task involving substantial numbers 
of staff and students, and the investmem of considerable resources. 

Because of the importance attached to the results of outcomes assessment studies, and the effort required 
to both initiate and complete them, it is cnjdal that institutions conduct thorough and precise outcomes 
assessment research. TTiis documem is intended to provide guidance in designing effective outcomes 
assessment research involving the GAAP. Following the procedures described here will significantly 
increase the likelihood of obtaining accurate and meaningful results. 

Soma Gwwfal ConsldaFatlons In Using Test Scorn 
Justfflcatton for Using Test Scores 

Outcornes assessment describes the results of students' experiences in a postsecondary educational 
institution. In general, outcomes assessment encompasses affective characteristics (such as students' 
interests and satisfaction) as well as academic characteristics. On the academic side, one of the goals of 
outcomes assessnrjent is to provide infomiatton that will be helpful in improving the academic perfomianca 
of students and faculty. 

Before deciding to use test scores in outcomes assessment, institutions must detennine whether test scores 
will provide them with the infonnation they are seeking. Test scores can be useful in outcomes 
assessment, provWed that they are appropriate measures of the outcomes of interest. For example, if an 
institution is interested in measuring students' pertomiance in mathematics, then there should be a 
sufficient match between the content of the mathematics curriculum and the content of the selected test. 

One way in which test scores may be used in outcomes assessment is to measure the relative academic 
pertormance of students. For example, test scores couW be used to measure the academic pertomiance 
of students in a particular college or university relative to criterion descriptions of pertomiance, or compared 
to nonns for a relevant reference group. If, for instance, the average test score for sophomores at 
University A is tower than the average test score for sophomores in similar programs at other comparable 
universities, then this may be an indicatton that University As sophomores have not reached similar levels 
Of academic proficiency. A group's level of academic proftoiency, however, will usually depend on its level 
of academic preparation on entering the program, as well as on other characteristtas. 



Another example of how test scores may be used in outcomes assessment is to measure the average 
amount of change occurring over time for studeirts in a given program or department. (Note that the 



emphasis hera is on group change; ACT does not recommend using GAAP to measure Individual 
change.) The average change is assumed to reflect the effectiveness of postsecondaiy instnjctlon. For 
example, if the average test score of students who have completed an instSution's core curriculum of 
general education courses is higtier than the average test score of a con^wrable group who have had no 
exposure to the core curriculum, then this couid be considered evidence of that program's effectiveness 
in educating them. The amount of gain may, of course, depend on the level of academic preparation 
students had on entering a program, as well as on other student characteristics. 

Using Test Scores Prooertv 

Test scores should not t)e the sole determinant for maldng decisions for improving curriculum effectiveness. 
Instead, they should be used in conjunction with other types of data, such as retention and graduation 
rates, course grades, and opinion survey responses. 

Decisions regarding the allocation of resources or funds within institutions, and decisions regarding the 
hiring, promotion, or retention of staff are of particular irrportance, and require the use of other types of 
data in conjunction with test scores. It would be inappropriate, for example, to allocate funds to a particular 
department solely on the basis of Its students' pre- and posttest scores. Lilcewise, the English department 
chair, for instance, should not feel that his or her department's position will be jeopardized solely because 
of a low average test score eamed by its students. 

In addition, low test scores are appropriately used only as indicators that further investigation should be 
condiicted. By themselves, such scores are not indisputable evidence of a problem with a certain 
curriculum or program, nor do they constitute a complete or effective outcomes assessment. 

Developing Formal Plans 

It is important for institutions to develop formal plans for outcomes assessment. Formal plans allow an 
institution, among other tilings, to avoid wasting financial resources and staff time. In the planning stage, 
an instftution engages in numerous activities. It will, for example, want to determine the assessment 
questions to be investigated (e.g., "Does our general education curriculum contribute to an increase in the 
skills and knowledge we want our students to acquire?^, and review any assessment data that have been 
collected prevtously by various campus departments and offtaes. These are only a few examples of 
activities involved in outcomes assessment planning; for more information, see College Assessment 
Planning (I990b). 

Another activity included in planning an assessment is the selection of appropriate measures of the 
outcomes of interest, tf an institutfon wants to determine the suitability of GAAP scores for its outcomes 
assessment, then the following suggestions may be helpful: 

1 . Devekjp an explicit description of the institution's goals for devetoping students' foundational skills. 
For example, an instltutton may want all of its students to demonstrate prof toiency in intermediate 
algebra by the time they have completed their sophomore year. 

2. Develop a listing and descriptton of the courses in the core academk; program being evaluated. 
There shoukJ be a logical relattonship between the institutton's goals for devetoping fundamental 
skills and the contents of the courses in its core academte program. For example, for sophomores 
to demonstrate proftoiency in intennediate algebra, they must take course wori^ in algebra, as part 
of the core academic program, that will enable them to sufftoiently develop the requisite 
mathematical skills. 



mo 
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3. Review the content specSications for ail of the GAAP tests to determine whether the institution's 
eyaiuation goals are likely to be met by using GAAP tests. This will help to avoid potential 
difficulties after the assessment is underway. l=6r example, programs lasting less than one year 
may be too short to be property studied with the GAAP. 

To assist institutions in identifying the match between test content and courses. AGT can furnish a specially 
developed iom. The forni describes the content of each of the GAAP tests; next to each description could 
be listed potential courses in the core program for which this test is viewed as appropriate. 

DevatoDlng a Testing Plan 

Once an outcomes assessment plan has been adopted that includes the use of GAAP tests, the institution 
should develop a testing plan. The testing plan should include: 

1. An explicit specification of the reference group(s) to be tested, and how they relate to the core 
program being evaluated. For exarriple. you might specify that you will test end-of-year 
sophomores who have nearly completed your institution's general education core cumculum. (See 
the following section.) 

2. A description of the sampling method, if any. to be employed. (See the section on below selecting 
and recruiting a sample.) 

3. The steps to be taken to inforni and motivate faculty and students about the testing. (Infomiation 
on motivating students is provided in a separate section below.) 

4. The GAAP modules to be administered and a description of how the tests are relevant to the 
programs being evaluated. 

5. The date{s) on which GAAP will be administered. 

6. A description of what testing facilities will be provided (e.g., our testing will take place in certain 
classrooms, we will have no more than 50 students per classroom, our faculty will serve as 
proctors, etc.). ' 

AGT has a checklist available for instituttons to use in devetoping their plans. 

As part of the testing plan, the institutton shouW develop a description of other instmments or indicatore 
It plans to use in its outcomes assessment. Retention rates, courses taken and grades earned, graduation 
rates, or responses to an opinion questtonnaire are examples of indicators that might be used This 
descnpton should indude consideratton of how these indfcators might be linked or integrated to draw 
conclusions and make dedstons about a program. lu ur^w 

The institution shouW also state the general kinds of decistons that will be made based on the results of 
the outcomes assessment. For example, if the results indteate that students' average pertonnance in 
mathemaics is unchanged from the freshman to the sophomore year, then the institutton couW plan to fomi 
a committee to investigate the content of certain mathematics courses. 

It may also be helpful to speculate about actions that might be triggered by results of various kinds in the 
Jr^U*^^' Mai*iemattos Department faculty may become defensive or feel threatened" by the 
?I!JPP°"2"^ ^f^^Ji stage of the outcomes assessment, it is explained to hem 

fr^f departments) that results will be used only to initiate further discuSion, and rSt to 

find fault with a given program, then their anxiety may be lessened 



ACT strongiy encourages all institutions to follow these protocols and docun^ient their testing plans in writing 
to ACT in advance of anticipated testing. As part of our outcomes assessment sen/ice, ACT staff will then 
review the plans and, if necessary, offer suggestions to help maximize the value of research findings. 

SAlwUng a Raferanca Group for TMtIng 

The reference group is the (sub)population of your students that you wish to study, and atsout which 
inferences wiU Ise made. A reference group can be any one of a number of intact groups that are logically 
related to an institution's goals for general education. One example of a commonly selected reference 
group is all sophomores who have completed the general education requirements at a particular institution. 
By testing these students, an institution could collect data pertaining to tlie overall effectiveness of its 
general education core cunriculum. 

Ottier examples of reference groups include: all sophomores with at least 60 semester hours of credit, or 
all sophomores enrolled in an institution's engineering program, or all students majoring in English who 
have completed at least 15 semester hours of English course work. 

The reference group selected must be logically related to the core program being evaluated. Further, the 
reference group and the core program must be compatibie with the institutions' goals for developing 
fundamental skills. For example, if an institution wishes to ensure that all of its graduates can write 
grammatically conect, and ogkally coherent essays (a fundamental academic sWll) then it might benefit 
most from evaluating its required courses in English composition taken by aU students during either the 
freshman or sophomore year. In this case, ttie reference group must consist of students who had recently 
completed, or nearly completed, the required courses in English composition. 

The selection of a test is dosely related to the selection of a reference group; ttie .est that is ultimately 
chosen must be appropriate for the reference group and the program being evaluated. An institution may 
find its assessment efforts wasted if the selected tast is not congment with the institution's core academic 
program and goals for devetoping students' basic academic skills. Selecting the CAAP Mathematics test 
to evaluate sophomores who have neariy completed tiie general education curriculum, for example, might 
be illogicai if the institution does not have a minimum profJdency in mathematics as one of its 
developmental goals for students, and does not require students to take course work in mathematics. On 
the other hand, an institution might wish to determine whettier it should add profidency in mathematics as 
an academic goal. One kind of infonnation that would be useful in making this deciston would be data 
comparing tiie inshtutton's students' mathematics skills with those of students at other institutions. 

Gftnttfaiiyj ^biHtv Of Outcomos Asaessmont Resuita 

The selection of a reference group has impUcations for the ways in which tiie results of an outcomes 
assessment can be used. The results of an outcomes assessment apply only to the reference group, and 
cannot be generalized to otiier reference groups. For example, if it is found tiiat completing certain 
coursewoik in mathematics increases tiie performance of engineering students on the CAAP Mathematics 
test, then this result is applicable only for engineering students; it may or may not be true tiiat the CAAP 
Mathematics perfonnance of sodology majors wouW be enhanced if they were to complete the same 
certain mathematics courses. 



SftlActlng and Racrutting an Ap^i'opriata 
Sampio of Students 



Tostlnoa >ODulatlon 



When poss ibie. it is best for an institution to test ail members of a reference group population. This method 
ts less tme-consuming than randomly selecting a sample of students, and it reduces the problems 
sometimes encountered when attempting to generalize findings from a sampie to a larger reference group 
population. When selecting students in this manner, it is important that all or nearly all of the reference 
group population actually take the GAAP, if only part of the population takes the GAAP, then the resulting 
subpopulaton can be considered a nonrandom sample. Administering the GAAP to nonrandom samples 
couW have senous implications for the results of an outcomes assessment: in some cases, the results will 
be uninterpretable. 

Sometimes, based on economic or other considerattons. an institutk5n will decide not to test all members 
of a reference population, but instead to sample from the population. Of the many considerations involved 
in conducting an outcomes assessment, few are as important as the selection of an appropriate sample 
Following are some examples of acceptable and unacceptable methods of sampling. 

Acceptable Methods of Sampling 

One ^ the ways an institution can obtain an appropriate sampie is by randomly sampling individual 
sttjdents. For example, one student could be selected at random from the first N students listed on a raster 
of all students. The selection of this first student could be perfcined in several ways: one possible method 
reUes on a table of random numbers. A number K between 1 and N is chosen from the table (e g the 
table IS opened to any page and the first number between 1 and N is selected). The Kth student is 
selected, and every Nth student is selected thereafter. untU a sufficient number of students is obtained. 

A poteritial difficulty in randomly selecting students is that sometimes they cannot be conveniently 
assembled for testing. For example, an institution located in an urban setting may select a sample that 
includes a significant proportfon of commuter students. Because these students are not often on campus 
relative to other students, it may be difficult to test the entire sample during a time that is convenient for 
everyone. If. as a matter of convenience, the commuter students are not included in the testing, then the 
sample will be biased and the outcomes assessment results will be applicable only to those students livina 
on campus. ^ 

A more practtaai method of sampling may be to select entire classrooms in whtah members of the reference 
^pul^ion are enrolled, and to administer the GAAP to all students in the class. For example if two 
S2^rc'^'^®t]Il^''®i!-'^? education curriculum are being evaluated, and they each have sbc different 
^"V"*.!!! ®^ '""'^ P"^® 3 sample of sufficient size. The random 
21 ^It^^"^ could also be perfonned using a table of random numbers. All students within the two 
selected sections would then be tested. 

^JoTJSm ^if ^T?^ 1. ^"^T^^ ^ ^'"•y important that all or nearty 

thin th« Tv!S"^ ^® ^u^- '* ^ '"^^^ only ^ small proportion of the sampie 

ttien the examinees may not be representative of ail students in the programs or courees beii^ ev^ted' 
Th«^ occur whether or not the initiai sample was represertative Forl^r^irSS sll^!^^^^^^ 
X randomly selected 100 of its 1.000 sophomore to take the GAAP. T^eleSira Ser w^ 
mandatory and only 20 sophomores chose to take the test. The group of 20 SiuTlS^ ^na^^ 
volumeers. and rn^ht differ from the typfcal College X sophomore (e.g.. they 4ht^ more 
For th« reason, those students who actually took the GAAP at Golle^e X n-20)^rS STreprosLrrtS 
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of all College X sophomores (N«1,000), even though they were randonrUy selected as part of the initial 
sample (n-lOO). 

Unaecentabla Methods of SamoHnq 

One mistake sometimes made when sampling is to select students simply on the basis of convenience. 
Let us say that, during the process of selecting a random sample of students, an institutional researcher 
finds that he or she has access to a list of sophomores who live In dormitories. Should the researcher 
select the sample biased on this incomptete list of students, or should he or she wait unta a complete list 
becomes available? Obviously, to select students from an frKXwnplete list could prevent the findings from 
being generalizabie beyond the study, and for this reason should not be done. 

Samples of students can be selected in other temptingly convenient (but eq iy unacceptable) ways. For 
instance, it would be inappropriate to select a convenient sample of students .ram one academic major or 
program (e.g., nursing majors) if other programs were also being evaluated. 

A totally unacceptable method of sampling is simply to ask for volunteers from the reference population. 
This method is appealing because it typically requires less effort than random sampling. Its main 
shortcoming is that volunteers will often differ from nonvolunteers with respect to important characteristics 
(e.g.. motivatton), thereby preventing the results of the study from being generalizabie to all students in the 
reference populatton. 

Additional Considerations in Salectinq a Sample 

The sample size needed for accurate inferences about a reference group will depend on many factors fiat 
cannot be controlled in advance, such as the size and academk: skills of the reference group being studied 
and the analyses being done on the resulting test scores. Therefore, it is not possible to specify a general 
rule for sample size that will be appropriate for ail situattons. 

ACT recommends as a "rule of thurrto' a minimum sample size of 100 students per reference group. 
Because the CAAP is modular, (i.e.. one or more of the five different objective tests and the essay test can 
be administered, rather than the entire test battery) this sample size recommendatton must be foltowed for 
each CAAP test that is administered. For example, an instftution wiU likely have sufficient data if it 
administers the entire CAAP battery, or 1 or more of the CAAP tests to each student in a sample of 100 
students from a given program. On the other hand, if each of 5 different groups of 20 students in this 
program takes a different CAAP objective test (which still results in a sample of 100 students being tested) 
then sample sizes (per test) will be too small to be useful. 

Institutions with small enrollments cannot always select a sample of 100 students per reference group. If 
an institutfon's reference group contains between 50 and 100 students, the institution should consider 
testing the entire reference group. If the reference group contains fewer than 50 students, then the 
institution shouU test students over a period of several years. untU a pooled san»le of sufficient size can 
be otrtained. 



Selecting an appropriate sample of students is a crocial aspect of any outcomes assessment Prior to 
receiving CAAP testing materials, each instltutton will be asked to provkle ACT with a brief written 
description of its selectton method and expected sample size. If you have questtons about selectinq a 
sample, or obtaining a sufficient sample size, please contact ACT. 



Motivating Students 



Atter an appropriate sample has been selected, another Important task in the outcomes assessment 
process is to motivate students. Students must not only be motivated to take the GAAP, but be motivated 
to perfonn their best as well Each institutfon can best decide what Is most effective in motivating its 
students. FoUowing are some suggestions. 

Methods Associated With Student and Instttutlonai Beneffls 

Students may be motivated to take the test and do their best if the benefits of the assessment, for students 
and the Institution, are clearly communicated. Students' motivation could be enhanced, for instance, by 
emphasizing that the assessment study provides a way to obtain infonnatfon about the quality of the 
education students have received. For this method to be most effective, an institution should clearly and 
consistently communicate to the students its commitment to outcomes assessment, and should make 
available to the students the overall results of the study. In additkjn, it may be helpful if students' 
partteipation in the assessment is made to appear as a nonnal. routine part of attending the institution 
rather than a special project requiring extra effort. 

Another way to improve student motivation is to infomi students that their participatran will benefit future 
students through program improvement. Moreover, such improvement couU lead to an increase in the 
vahje of students' degrees. 

Methods Associated WHh Rewarding Students 

Directly rewarding students can be an effective method of motivation. Examples of substantive privileges 
that can be given to students on the basis of their participation and/or pertomiaiice include: 

1. Paridng in desirable locations on or off campus. 

2. Receiving eariy course registration privileges. Avokjing long registration lines, or having relatively 
easy access to certain very popular courses may be Incentives to some students. 

3. Having first chance at buying tickets to concerts, athletto events, etc. 

4. Moving to the top of the waiting list for a room in a desirable residence hall. 

Meaningful finandai incentives, awanJed to students on the basis of their participation and/or perfonnance 
can also sen/e to increase student motivatfon. Examples inckide: 

1. Reduction in tuition. 



2. Gift certiftoate from the college or univeroity bookstore. 

3. Tickets to a popular event that will be held on campus in the near future (e.g., a concert). 

^® ^ °^ P'a^s can be made regarding the method for 

SL^J^nS"*!^,^*.^^ r^"*^'^' ^""^^^ "'^^ to all ^dent/who scored o 

above a given cutoff. Another possibility is to enter the names of students scoring above a particular score 
into a totteiy system, whereupon several winners will receive awards such as those listed above. 
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Rewarding students for increases in test performance may also be an effective form of motivation. For 
example, an award could be made if a student's GAAP score obtained at the completion of a program is 
high relative to a related test score obtained before any exposure to the program. This method is an option 
only for those institutions that perform longitudinal testing of students or that wiU, in the future, use students' 
ACT Assessment or ASSET scores obtained e* entry in conjunction with students' end-of-sophomore year 
CAAP scores to anive at an estimate of change (ACT is currently designing such a system). ACT can 
assist institutions in implementing a method of rewarding students for score increases. 

tt is possble to reward students merely for their participation in the outcomes assessment, independent 
of their perfonnance on the CAAF, but this may faM to motivate them to perfonn their best. For example, 
an institution may offer free t-shirts to students if they complete the test If students are interested only in 
turning in a completed answer sheet to otrtain the award, and not interested in attenrpting to achieve the 
best possa)le scores, then the results of the test will be less valid. 

Mandatory Testing 

Another method for motivating students is to make the testing mandatory. For mandatory testing to be 
effective, however, it is important that students try to do their oest on the CAAP, rather than simply being 
present for the testing. If students are not sincere in their efforts, then their scores will not be valid for use 
in outcomes assessment. 



One suggestion for encouraging students to perfomi well during a mandatory administration of CAAP is 
to require them to achieve a certain minimum score or higher in order to regfeter for any future courses. 
The minimum score could be set tow enough so that only those studer^ vho deliberately put forth no effort 
(e.g., marking answer •B" for every test question) wouW be penalized. In a more stringent application. 
cutofJ scores could be set high enough so that only students have high estimated chances of success 
wouW be pennitted to take upper^evel course work. If cutoff scores are to be used in this fashion, then 
it is important that the institution conduct a local valWlty study to make certain that its selected minimum 
score is appropriate, and that students are not penalized unnecessarily. All sUidents, moreover shouW 
be given the opportunity to take the test again and improve their scores. 

Another suggestion is to make students' CAAP stores a permanent part of their academic record. One 
potential disadvantage of this method is that ii may be perceived by high-scoring students as rewarding, 
but be perceived by low-scoring students as threatening. 

Soma Additional Consldarations 
In Admtnisterlna the CAAP 

The concept of the validity of test scores applies not only to the uses made of them, but to the entire testing 
process as well, including the administration and scoring of the test If the test is improperty administered 
then the test scores cannot be valW. Consider, tor example, what wouW happen if the test were 
administered under nonstandardized condittons to students in different programs, departments, or 
instituttons. Any comparisons of mean scores among such groups, or comparison with other groups who 
took the test under standardized condittons, woukl be inappropriate. An institution woukJ find it difffeult to 
justify, for Instance, the companson of average scores between two departments, one of which 
administered the test in a quiet classroom under standardized condittons, while the other tested studer?ts 
in a crowded cafetena and used a test supennsor who neglected to read part of the instmctfons for the test. 

To ensure that the best quality data are being obtained, and that valid uses can be made of the test scores 
^'J' r!f?^^ ^ instituttons strictly foltow the standardized test administratton procedures described irl 
the CAAP JesL^upen^r's Manual (1990a). In addition, we advise using at least one proctor for every 
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25 students tested. Proctors, like the test supervisor, should t>e familiar with the corrterrts of the manual 
prior to the administration of CAAP. 

Test supervisors are sometimes the first to identify protjiems with students' answer sheets. Before sending 
completed answer sheets to ACT, it is a good idea for the test supervisor to examine them for patterns 
An unmotivated examinee may. for example, use the drctes on the answer sheet to create objects, designs 
or messaeas. tf patterned answer sheets are found. ACT recommends that the test supervisor copy from 
them the students' names, SSNs, and test booklet numbers. This infonnatton can be recorded on a 
separate paoe, and attached to the completed Irregularity Report ACT wiU not score these answer sheets. 

Concluston 

The purpose of this document is to offer suggestions for designing and conducting effective outcomes 
assessment research involving the CAAP. By adhering to the preceding conventions, institutions will 
greatly increase their chances of obtaining accurate and useful results. 

We hope that the infonnation presented in this document will be helpful in you. outcomes assessment if 
you have questions, or need additional guidance, please contact ACT. Our staff will be happy to assist 



ReferencM 

The American College Testing Program (1990a). CAAP Test Supervisor's Manual, lowa City Iowa- 
Author. 

The American College Testing Program (1990b). College Assessment Planning , lowa City, Iowa: Author. 



For assistance with CAAP, contact 
Dr. David A. Lutz 

Director, College Level Assessment and Survey Services 
American College Testing 
P.O. Box 168 
lowa City, lA 52243 
Tel. 319/337-1051 

OR 

Dr. Jeff Schiel 
Research Associate 
Tel. 319/337-1076 
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ESS 



ACT Evaluation/Survey Service 



(Prices effective September 1, 1992->Subject to change without notice) 

To order ESS materials, you must use this form even if you submit a purchase order. Place your order 
at least three vveeks before you need the materials. Materials will be shipped by UPS or comparable 
method. Please print or type. 



Description 



Quantity 



Price 



Survey Instruments i Distributed in packaRes containing 25 instruments 
and, if appropriate. 25 copies of the major/occupation code list.) 

1. Adult Learner Needs Assessment Survey (package of 25) 

2. Alumni Survey < packaJ?e of 25) 

3. Alumni Survey (2-Year College Form) (package of 25) 

4. Alumni Outcomes Sur\'ey tpackapo of 25) 

5. College Outcomes Survey (package of 25) 

6. College Student Needs Assessment Survey (package of 25) 

7. Entering Student Sur\*ey {package of 25) 

8. Student Opinion Survey (package of 25) 

9. Student Opinion Survey {2.Year College Form) ^package of 25) 

10. Surv'ey of Academic Advising i package of 25) 

11. Surrey of Current Activities and Plans ( package of 25 J 

12. Sur\-ey of Postsecondnry Plans 'pat.'kai:e (»f 25) 

1.^. WithdraunngNonureturning Student Sur\'oy •packa;;e of 25) 

14 WithdraunngNonreturning Student Sur\-ey (Short Form) 
•pack 3 CO 1 \ 

ESS Specimen Sol 

(.n.,\jdc> •■Hi' c'.py < U ach .<urvcv insininu-nt. an E>S ..rd- r { -rm. 
srvt-rai >amr../ rep -rt pauvs. a samr,"k» Snhjr up S-K in. n Form, and 

ES.S I'mt s (iuido 

(r>c^ with orr:er oj'4«!'» or rr.-Tr ^urv.-y iri£trin>nt5, als-. inuudt-d 
■•vuh the H;s.S Sretimrn ah w 

KSS iit n) t'jitulDg 

(('onth'.!;- several r-.-jniirr i ^Amrl^ >Jonv- to • cii ,.'ir ii >t.t'.jti'.n 
tor stn:u its own I0^'«^l Hur\»»y •v.j'">-ti ns) 



pkgs. 
pkgs. 
pkgs. 
pkgs. 
pkgs. 
pkgs. 
pkgs. 
pkgs. 
pkgs. 
pkgs. 
pkgs. 
pkgs. 
pkcs 
pkgs 



$7.50 
$7.50 
S7.5U 
$7.50 
$7.50 
$7.50 
$7.50 
$7.50 
$7.50 
$7.50 
$7.50 
$7.50 
$7 .V) 



00 



Total 



ERIC 



05 



hi RTOTAI. 



Normative Data Reports (Each report contains comparative data 
bai>ud on student records from colleges that administered the 
particular surve> instrument(s).) 



1, Adult Learner Needs Assessment Survey Normative Data Report 




$25.00 


$ 


2. Alumni Survey Normative Data Report 


cnpies 


$25.00 




3. Alumni Survey (2-Yr. College Form) Normative Data Report 


rnpinQ 


$25.00 




4. College Student Needs Assessment Survey Normative Data Rpt. 


cnpies 


$25. {)0 




5. Entering Student Survey Normative Data Report 


ropif^S 


$25.00 




6. Student Opinion Survey Normative Data Report 


rnpipQ 


$25.00 


% 


7, Student Opinion Survey (2-Yr. College Form) Normative Data Rpt. 




$25.00 


$ 


8. Survey of Academic Advising Normative Data Repoi-t 


rnpip«i 


$25.00 




9. Withdrawing/Nonretuming Student Survev Normative Data Rpt. 


rn pi PQ 


$25.00 




10, Withdrawing/Nonreturning Student Survey (Short Form) N.D.R. 


rnpip«i 


$25 0(1 


$ 




Subtotal from side 1 


$ 


(California residents add 67r sales tax.) 




TOT^VL 


$ 



\f ti>tal is less than $15.00, please enclose payment with firder. 
Postage and shipping charges Vrill be added to all invoices. 

ACT Scoring/Reporting Service 

As part of the standard ESS program, ACT will optically score your institution's completed survey 
instruments and prepare a summary report of the results, (Scoring/reporting prices: $80.00 
reporting/liandling fee or $100.00 reporting/handling fee for a laser printed report plus $,60 per 
student record processed.) The sumn^qry report is based on up to 15 student subgroups of your choice 
and provides extensive frequency data for all items on the particular sur\'ey. Institutions that use the 
Scoring/Reporting Service may also obtain copies of their student data on magnetic tape or IBM 
compatible PC diskette. 

If you plan to administer an ESS instrument do you also plan to use the ACT Scoring/Reporting 
Ser\ice? 

n Yes □ Undecided No 
Ship Materials To: Bill To: 



Datp 



ERIC 



Return ihi< I'.rm to: 



ACT 

Ev.iiu.-iiii n >\\r\, \ r\ »• Cfe'O 

P O B,.x 168 

low.i City^ lov^n 522*3 



